Viewing a virtual reality environment on a user device

ABSTRACT

A system for augmented reality viewing generates a unique code corresponding to a user device. Information corresponding to the unique code is broadcast from the user device and the unique code is displayed on the user device. The information is configured to be received by an augmented reality device. A camera of an augmented reality device is activated upon receiving the information broadcast from the user device. The camera is configured to scan for the unique code. The user device is joined to an augmented reality session of the augmented reality device in response to the camera scanning the unique code displayed on the user device and by transmitting identification information to the user device based on the unique code. The simplified joining process improves the user experience and allows the user to more quickly view an augmented reality environment directly on a user device.

CROSS-REFERENCE TO RELATED APPLICATION

This nonprovisional application claims the benefit of and priority toU.S. Provisional Application No. 62/673,808, filed May 18, 2018, whichis incorporated by reference herein in its entirety.

BACKGROUND

Augmented reality devices allow a user to view a real-world scenethrough a lens with holograms appearing as part of that real-worldscene. Augmented reality and virtual reality systems are increasinglypopular, particularly for applications in which a user can immersehimself or herself into an alternate reality environment when wearing ahead-mounted display that displays virtual and/or augmented reality userexperiences.

For example, augmented reality allows a user to see through ahead-mounted display, such as glasses or other wearable display deviceshaving near-eye display panels as lenses to view the surroundingenvironment, while also seeing images of virtual objects. These virtualobjects often include holograms that are generated for display andappear as a part of the environment. Thus, augmented reality systemsdisplay various types of virtual images or graphics that enhance oraugment the environment that a user experiences.

While some known augmented realty devices allow for recording theaugmented reality environment being viewed by the user, the recordingcan only be done from the viewpoint of the user of the augmented realitydevice (e.g., the wearer of the head-mounted display) and is typicallyacquired at a lower resolution. In order to view or record the augmentedreality environment from a different perspective, such as with a devicethat can view the augmented reality environment, but record at a higherresolution, a time-consuming setup process of the device is performed.For example, the setup process involves extensive arrangement andorientation of the device(s) for recording, as well as synchronizing thedevices with the augmented reality system to allow for proper recordingof the augmented reality environment. Thus, there is often no easy wayto view and record augmented reality scenes using devices that are notthe primary scene generator.

SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

A computerized method for viewing an augmented reality environmentcomprises generating a unique code corresponding to a user device anddisplaying the unique code on the user device. The computerized methodfurther comprises broadcasting information formatted for reception by anaugmented reality device. The broadcast information is based at least inpart on the unique code. The method also comprises automatically joiningthe user device to an augmented reality session of the augmented realitydevice in response to the augmented reality device scanning the uniquecode displayed on the user device. At least one hologram in theaugmented reality session is spatially aligned to a view of the userdevice

Many of the attendant features will be more readily appreciated as thesame becomes better understood by reference to the following detaileddescription considered in connection with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The present description will be better understood from the followingdetailed description read in light of the accompanying drawings,wherein:

FIG. 1 is an a block diagram illustrating a system according to anembodiment;

FIG. 2 illustrates an initialization process for joining a user deviceto an augmented reality session according to an embodiment;

FIG. 3 illustrates an authentication process for joining a user deviceto an augmented reality session according to an embodiment;

FIG. 4 illustrates an authentication code according to an embodiment;

FIG. 5 illustrates a joining a user device to an augmented realitysession according to an embodiment;

FIG. 6 is a flowchart of a process for viewing an augmented realityenvironment according to an embodiment; and

FIG. 7 illustrates a computing apparatus according to an embodiment as afunctional block diagram.

Corresponding reference characters indicate corresponding partsthroughout the drawings. In the figures, the systems are illustrated asschematic drawings. The drawings may not be to scale.

DETAILED DESCRIPTION

The computing devices and methods described herein are configured toallow viewing and recording of augmented reality environments from userdevices in a shared experience session with the augmented reality devicegenerating the augmented reality scene. For example, remote viewing andrecording of the augmented reality environment (that includes one ormore holograms) is easily accomplished by a user device in accordancewith the present disclosure. In one example, an augmented realitycapture tool allows smartphone users to see what Microsoft HoloLens®device users are seeing and doing.

Some configurations described herein allow for an image stream that ishigher resolution and lag free to be made available by a simple processfor inviting non-HoloLens device users to experience holograms directlyfrom the user's phone or tablet. As a result, a faster and moreefficient setup process for viewing augmented reality scenes is providedthat improves the user experience. In this manner, when a processor isprogrammed to perform the operations described herein, the processor isused in an unconventional way, and allows the more efficient setup andviewing of an augmented reality environment by a user device. Moreover,in various examples, with the user devices, recording of the augmentedreality environment can be performed at a higher resolution than thatwith the recording capabilities of the augmented reality device viewingthe augmented reality scene.

In some examples, a person can wear a head-mounted display unit toimmerse himself or herself in a virtual and/or augmented realityenvironment. Generally, the term “alternate reality” is used herein torefer to devices and systems that are implemented for virtual realityand/or augmented reality. A head-mounted display unit is an alternatereality device that can be worn by a user and implemented with varioussystems and sensors to generate a hologram (e.g., a generated image orgraphic) that appears in an environment with real objects (e.g., aHoloLens® device). Various examples can be implemented in connectionwith any type of device capable of operating in a virtual and/oraugmented reality environment.

For example, an alternate reality device is implemented with variouscomponents in the virtual and/or augmented reality environmenttechnology, such as a processing system and memory (e.g., non-volatile,physical memory), and with any number and combination of differingcomponents. In some examples, the processing system includes multipleand/or different processors, such as a microprocessor, a graphicsprocessor, and/or a high-speed, dedicated processor for tracking motionof the alternate reality device. The alternate reality device includes apower source, such as a battery, to power the various device components.Further, the alternate reality device in some examples is a wirelesscommunication-enabled device and can include different wireless radiosystems, such as for Wi-Fi, Bluetooth®, Mobile Broadband, LTE, as wellas 802.11a/b/g/n/ac network connectivity technologies, and/or any otherwireless communication systems or formats. Generally, the alternatereality device implements one or more wireless communication systemsthat each include a radio device, antenna, and chipset that isimplemented for wireless communication with other devices, networks, andservices.

The alternate reality device can include a tracking system for entitytracking that is implemented to recognize and track movement of anentity in the environment. For example, the tracking system canrecognize an entity in the environment as a person or features of theperson (e.g., an arm, hand, head, etc.), an animal (e.g., a dog or cat),or an object, such as a chair, a table, or other item that can be movedby a person. The tracking system also includes in some examples, amotion sensing or any other human motion capture system to track themovement of the person or the feature of the person in the environment.

Additionally, an imaging system, such as with an infra-red projector, adepth sensor, and/or a depth camera (or cameras) allows capture ofimages of the environment in which the device is located. The imagingsystem of the alternate reality device in some examples has one or morecameras that capture images of the environment in which the alternatereality device is being used. In various examples, the cameras arevisual light cameras, such as high-speed monochromatic orblack-and-white cameras that capture the images in the three-dimensional(3D) environment.

In various examples, the alternate reality device is any type of displaysystem providing a view through optics so that the generated image thatis displayed to the user is overlaid onto a real-world view. Thisaugmented reality or equivalently mixed reality is provided in someexamples by the HoloLens® device. Thus, as a wearable display system, anaugmented reality device can incorporate components, such as processingunit(s), computer interface(s) that provide network connectivity, andcamera(s) etc. These components can be housed in the headpiece, as inthe HoloLens®, or in a separate housing connected to the headpiece bywireless or wired means.

The alternate reality device also includes an imaging application thatis implemented to generate holograms for display. The imagingapplication can be implemented as a software application or modules,such as computer-executable software instructions that are executablewith the processing system. The imaging application can be stored oncomputer-readable storage memory (e.g., the memory), such as anysuitable memory device or electronic data storage implemented in thealternate reality device.

FIG. 1 is a schematic block diagram of a system 100 deployed in a WiFicommunication configuration in this example that allows one or more enduser computing devices, such as a smartphone 102, a laptop computer 104and/or a tablet 106 to experience an augmented reality environment(e.g., experience holograms) of an augmented reality head worn computer108 (e.g., HoloLens® device). A user is able to experience hologramsdirectly from the end user computing devices by entering a sharedexperience session with the augmented reality head worn computer 108. Inthe illustrated example, the end user computing devices and theaugmented reality head worn computer 108 are connected via a computernetwork 110, shown as a WiFi network. For example, the user of theaugmented reality head worn computer 108 is able to invite non-HoloLensusers, namely the users of the smartphone 102, the laptop computer 104and/or the tablet 106 to experience holograms directly from the end usercomputing devices. In some examples, the non-HoloLens users, namely theusers of the smartphone 102, the laptop computer 104 and/or the tablet106 request to join the augmented reality session of the augmentedreality head worn computer 108. In some examples, the shared sessionexperience allows the smartphone 102, the laptop computer 104 and/or thetablet 106 to view holograms and record the augmented realityenvironment of the augmented reality head worn computer 108.

The computer network 110 in some examples is also connected to one ormore computers 112 and storage 114, which can be networked computers orstorage devices. It should be noted that in various examples, all of theaugmented reality functionality is implemented within the augmentedreality head worn computer 108 and the end user computing devices haveall of the functionality therein to allow for joining the augmentedreality experience. In some examples, the functionality of the augmentedreality system described herein is performed, at least in part, by oneor more processors or hardware logic components within one or moredifferent components or devices. For example, and without limitation,illustrative types of hardware logic components that are used includeField-programmable Gate Arrays (FPGAs), Application-specific IntegratedCircuits (ASICs), Application-specific Standard Products (ASSPs),System-on-a-chip systems (SOCs), Complex Programmable Logic Devices(CPLDs), Graphics Processing Units (GPUs).

In one example, the system 100 allows for automatic interfacing with theaugmented reality head worn computer 108 by one of more of the end usercomputing devices. As described in more detail herein, using anauthentication code or marker, such as an augmented reality marker (ARmarker), a simplified authentication and connection arrangement isprovided. Thus, in various examples, a marker-type augmented realityconfiguration is used, wherein the AR markers or other visual cuestrigger the automatic connection to a shared experience session. Itshould be noted that the marker can be any type of visual indicator andin some examples is displayed on the end user computing device during anauthentication and joining process. For example, a camera of theaugmented reality head worn computer 108 is configured to recognize anAR marker (e.g., recognize the geometry by identifying portions of themarker) displayed on one of the smartphone 102, the laptop computer 104and/or the tablet 106 to automatically authenticate the smartphone 102,the laptop computer 104 and/or the tablet 106 and allow connection to ashared experience session. In the shared experience session, thesmartphone 102, the laptop computer 104 and/or the tablet 106 is able toview the augmented reality scene being viewed by the augmented realityhead worn computer 108.

As described in more detail herein, this authentication process alsoinitiates an alignment and orientation process to properly position theview of the end user computing device within the augmented realityenvironment (e.g., image registration). Thus, the coordinate system forthe end user computing device is aligned with the augmented realitycoordinate system of the augmented reality environment displayed by theaugmented reality head worn computer 108. It should be noted that invarious examples, the AR Marker is used to determine the location of theuser device in the HoloLens® device space (e.g., a position or locationof the user device is determined from the marker), which is parsed tothe user device. As such, the user of the user device is able to viewthe augmented reality scene as if the user is wearing the HoloLens®device, but from a different viewpoint.

A process for joining an augmented reality session is illustrated inFIGS. 2-5. The process can be implemented as part of an augmentedreality sharing tool that allows, for example, smartphone users to seewhat a HoloLens® user is seeing and doing, as well as to stream higherresolution (e.g., high-definition) augmented reality images of theaugmented reality environment (e.g., real-world view with holograms).

In the illustrated embodiment, the operations can be performed in adifferent order than is shown, can include additional or feweroperations and can be modified as desired or needed. Additionally, oneor more operations can be performed simultaneously, concurrently orsequentially. The process in one example includes:

1. At 200, the HoloLens® device (host) 212 adjusts a world container 214in a room at 216. That is, the HoloLens® device 212 hosts and starts asession (e.g., a shared experience session). The world container invarious examples is a virtual space defined within the physical room(e.g., on top of a table).

2. At 202, the HoloLens® device 212 places the world container 214. Forexample, the augmented reality system of the HoloLens® device 212defines an augmented reality space 218 in which augmented realityobjects (e.g., holograms) are placed, thereby allowing interaction withthe augmented reality objects. In the illustrated example, the aspherical hologram 220 is the augmented reality object. The HoloLens®device 212 thereby defines the augmented reality environment to beexperienced in the real-world scene.

3. At 204, a user 222 launches an application 224 (e.g., a sessionjoining application) on a user device 226. For example, a smartphoneuser (e.g., iPhone® user) launches an augmented reality application,which in some examples is implemented in part using an AR kit configuredfor the smartphone (e.g., iPhone® ARKit). The launching of theapplication in some examples initiates an augmented reality session. Forexample, when the application 224 is launched on the user device 226,the user device 226 broadcasts a request to connect with the HoloLens®device 212 to join the augmented reality experience of the HoloLens®device 212. The request is broadcast within a defined range of the userdevice 226, such as based on the particular access point of a WiFinetwork to which the user device 226 is connected. In the illustratedexample, the user device 226 and the HoloLens® device 212 are connectedto the same WiFi network. Thus, in one example, the user device 226broadcasts the request to all of the same application bundle ID sessionson the WiFi network. As such, other devices (not shown) within the sameapplication bundle session, such as all HoloLens® devices 212 arenotified of the request to join the session.

4. At 206, while the user device 226 finds one or more surfaces withinthe real-world (e.g., locates table, chairs and walls), the AR Kit onthe user device 226 performs plane finding operations. For example,using a camera of the user device 226 and orientation/position sensors(e.g. gyroscopes) of the user device 226, real-world surfaces in areal-world view are identified (e.g., table, chairs and walls), whileaugmented reality plane discovery operations are being performed usingplane finding techniques in the augmented reality technology. That is,planes within the augmented reality space having the world container 214are identified.

5. At 208, an authentication code 228 is then generated (based in parton the determined location and orientation of the user device 226) anddisplayed on the user device 226. For example, an AR marker 232 (asshown more clearly in FIG. 4) is displayed on a screen 230 of asmartphone. It should be appreciated that the generation and display ofthe AR marker is 232 performed automatically in various examples, as aresult of the launching of the application on the user device 226. Theuser device code, such as the AR marker 232, represents the ID for theuser device 226 to allow connection to an available HoloLens® device212. That is, the user device code identifies the device to theHoloLens® device 212 as a device of the user 222 that wants to join theaugmented reality session. When the user device code is generated, theHoloLens® device 212 is notified of the generation of the code. Forexample, the AR marker 232 details are transmitted to the HoloLens®device 212 (such as through the WiFi network) to allow the HoloLens®device 212 to recognize a device displaying the AR marker 232.

As a result of a HoloLens® device 212 being notified of the user devicecode, the HoloLens® device 212 automatically turns on a camera 234 ofthe HoloLens® device 212 and starts searching for the AR marker 232. Itshould be noted that the user 236 of the HoloLens® device 212 in someexamples is notified of the request received from the user device 226(e.g., receipt of the AR marker 232). For example, a notification isdisplayed in the lens 238 of the HoloLens® device 212 that the userdevice 226 has requested connection to the augmented reality session. Itshould be noted that the code can be any code recognizable by theHoloLens® device 212 and used to validate or authenticate the userdevice 226 to allow the user device 226 to join the augmented realitysession.

6. With the HoloLens® device 212 in the scanning mode, when any userdevice, including the user device 226, approaches the HoloLens® device212, the HoloLens® device 212 scans the user device 226 at 210, forexample, scans the display 230 of the user device 226. If an AR marker232 is displayed on the user device 226, the HoloLens® device 212 scansthe AR marker 232. If the AR marker 232 scanned matches the AR markerinformation received by the HoloLens® device 212 (i.e., the user device226 that broadcast the request to join the augmented reality sessions)then the user device 226 is joined to the session as described below.

In particular, in one example, the HoloLens® device 212 finds a code,such as the AR marker 232 and reads the ID (e.g., reads the ID of thatmarker). The HoloLens® device 212 is configured in some examples toprocess the AR marker 232 to identify defining characteristics todetermine if the displayed code is a match. In one example, theHoloLens® device 212 then broadcasts the ID, such as using the WiFinetwork (or other wireless broadcasting technology), which is receivedby the user device 226. The user device 226 then compares the receivedID to the ID generated by the user device 226 (e.g., the AR marker 232generated by the user device 226 when the application was launched). Ifthere is a match, the user device 226 then joins the augmented realitysession and a confirmation indicator 238 is displayed in some examples.For example, through the augmented reality functionality of the userdevice 226, a connection is established and the user device 226 joinsthe augmented reality session using augmented reality sharing techniquesin the augmented reality technology. It should be noted that thedetection and identification of the AR marker 232 is performed using anyAR marker detection process in the augmented reality technology.

7. The user device 226 then displays the augmented reality environmentas viewed by the HoloLens® device 212 at 240. That is, the augmentedreality functionality of the user device 226 displays any holograms (thespherical hologram 220 in this example) within the augmented realityenvironment (world container) of the HoloLens® device 212 in the exactsame plane as the HoloLens® device 212. That is, the hologram(s) arealigned in the coordinate system of the HoloLens® device 212. In oneexample, the alignment is accomplished by the HoloLens® device 212sending spatial information to the user device 226. Thus, it should beappreciated that in various examples, images are not being communicatedto the user device 226, but instead, data corresponding to the augmentedreality environment of the HoloLens® device 212 is displayed. Moreover,once the user device 226 is joined to the augmented reality session ofthe HoloLens® device 212, the processing and control of the view of theaugmented reality environment can be performed by the user device 226.

The augmented reality experience of the present disclosure allows theuser device 226 to view the augmented reality environment of theHoloLens® device 212 in a spatially aligned scene. That is, the userdevice 226 is able to view and capture the augmented reality scene ofthe HoloLens® device 212 directly from the user device 226 (e.g., useris a spectator within the augmented reality environment). In this modeof operation, the user device 226 is able to, for example, film orrecord high-definition holograms. That is, the user device 226 is ableto record holograms at the screen resolution of the user device 226,which in various examples is full high-definition. Additional imagingfunctionality of the user device 226 also can be performed, such as toapply antialiasing and shadowing functionality.

In some examples, with the user device 226 joined to the augmentedreality session, the user device 226 is able to perform live streamingof the augmented reality environment. For example, the user device 226is able to stream a live high-definition holographic experience to astreaming device lag free (e.g., stream to an Apple TV® device directlyfrom an iPhone®).

In some examples, while joined to the session, the user device 226 isconfigured to invite guests to the augmented reality session. Forexample, the user device 226 is configured to add additional devices toallow other non-HoloLens device users to experience holograms directlyfrom a phone or tablet (e.g., the additional devices each generate acode that is displayed and scanned by a camera of the user device 226connected to the augmented reality session to thereby authenticate theadditional devices to allow connection of the devices to the augmentedreality session). It should be noted that the invitation process can beperformed in different ways, including by directly sharing the augmentedreality experience, transmitting a session ID to join, etc.

Thus, in some examples, the HoloLens® device 212 places one or moreholograms and the shared experience of viewing the hologram(s) isinitiated by launching an application on the user device 212. Once theuser device 212 is connected to augmented reality session and viewingthe hologram, the HoloLens® device 212 operates as the server and theuser device 226 operates as the guest in one example.

In some examples, the present disclosure provides one or more of thefollowing features:

1. Network auto-discovery for adding user device (e.g., mobile phones)to the session.

2. Automatic session handling, so users are added to the correctsession.

3. Spatial synchronization of holograms, so all users see holograms inthe exact same place (i.e., spatially aligned).

4. iOS® support (ARKit enabled devices).

5. Multiple iOS® guests.

6. Recording of video, holograms, ambient sound and/or hologram sounds.

7. A sharesheet so a user can save the video of the augmented realityscene, email the scene, or share the scene with other supportingapplication.

In some examples, the present disclosure is implemented in connectionwith one or more different licenses, such as an OpenCV license (3-clauseBSD License) and a Unity ARKit license (MIT License). Additionally, oneor more of the components described herein operate with a plugin andOpenCV binaries (e.g., UnityARKitPlugin and associated binaries).

Different network discovery technologies can be used in accordance withthe present disclosure. In one example, UNET is used for networkdiscovery and spatial synchronizing. As such, all interactivity duringthe application is synchronized between the devices. In variousexamples, UNET controls networking and manages all host-clientconnections.

The examples disclosed herein can be used with different types ofaugmented reality or alternate reality devices. In one example, thepresent disclosure contemplates implementation using a HoloLens® deviceand a Windows® computing device running, for example, Windows® 10, incombination with an ARKit compatible device (e.g., iPhone 6s ornewer/iPad Pro 2016 or newer/iPad 2017 or newer) running iOS 11 orabove. However, as should be appreciated, other operating environmentsare contemplated, such as with an Android® operating environment. Otheroperational examples include a Mac® device with xcode 9.2 or newer. Insome examples, Microsoft® Visual Studio 2017 is implemented.

In one example, the augmented reality sharing arrangement describedherein is set up by:

1. Preparing a scene.

2. Ensuring all visible gameobjects, within the scene, are containedunder a world root gameobject.

3. Adding one or more prefabs according to the present disclosure to thescene.

4. Selecting a corresponding gameobject.

Thus, in some examples, using an application configured for the userdevice and augmented reality device, these devices can be connectedtogether in the augmented reality experience (e.g., a smartphone andhead-mounted augmented reality device connected to the same WiFinetwork). When the application is started on the devices (in no specificorder), such as on a user device operating on the same network as theHoloLens® device, upon launching the application, the camera of theHoloLens® device is turned on and begins acquiring images and the userdevice turns on a camera and looks for surfaces (e.g., floor or table).The devices are then connected using a marker that is displayed on theuser device and recognized by the HoloLens® device. The marker is nolonger displayed once the marker is detected by the HoloLens® device,indicating to the user of the user device that the user device isconnected to the augmented reality session of the HoloLens® device, andboth devices are spatially synchronized as described herein.

It should be noted that the different features can be implemented usingdifferent input parameters. As an example, to capture and save a videofrom the user device, the user taps and holds the screen for one second,which opens a recording menu. Tapping a record button (e.g., a redrecord button) starts a countdown before beginning to record the screen.To finish recording, the user taps and holds the screen for another onesecond and taps the stop button. By clicking a preview button (e.g., ablue button), a user is able to watch the recorded video on the userdevice. The camera roll can be saved in some examples by opening asharesheet.

It should be noted that application specific data is synchronized invarious examples, such as using SyncVars, NetworkTransform, orNetworkBehavior, among others.

Thus, various examples allow user device access to augmented realitysession that allows for streaming high quality, instant, third personviews of holographic experiences, which can be displayed on a largescreen (through a TV streaming device). With the present disclosure,multiple users can join the HoloLens® device experience by viewing theaugmented reality environment directly from the user device. As such,portable device viewing of the augmented reality environment is easilyaccomplished.

Variations and modifications are contemplated. For example, the presentdisclosure can be implemented in connection with different operatingplatforms (e.g., Android® support). Video adjustments (e.g., focus andwhite balance) can be performed. A simple mobile input (e.g., augmentedreality user interface) at the user device can be provided (e.g., phoneusers can interact with the augmented reality scene). Sound controls(e.g., mute, ambient and in-app sound) can be implemented. In someexamples, a screenshot (photo option) is provided that allows for simpleand fast screen capture. In some examples, longbow or other networksolution support is provided.

Additionally, in some examples, a calibration process can be performed.For example, using a calibration pattern (e.g., defined printedcalibration pattern), a camera of the user device scans the calibrationpattern to confirm image calibration using image calibration techniquesin the calibration technology.

The present disclosure can be implemented in different applications,such as:

Filming holograms in HD: Using various examples, a user can record amixed reality experience using an iPhone®, including recording in fullHD and apply anti-aliasing to holograms and shadows. Thus, acost-effective and quick way to capture video of holograms is provided.

Live demonstrations: Stream live mixed reality experiences to an AppleTV® directly from an iPhone® or iPad®, with reduced lag or lag-freestreaming.

Share the experience with guests: Let non-HoloLens users experienceholograms directly from the user's phones or tablets.

Thus, joining an augmented reality session, such as with a user device(e.g., smartphone), can be performed easily and quickly using variousexamples by:

1. Ensuring that both devices are on the same Wi-Fi network.

2. Starting the application on the both devices, in no specific order.The process of starting the application on the user device (e.g.,iPhone®) triggers the HoloLens® camera to turn on and begin takingpictures.

3. As soon as the user device application starts, the user device looksfor surfaces like floors or tables. When surfaces are found, a marker isgenerated and shown to the HoloLens.

4. Once the marker has been detected by the HoloLens®, andauthenticated, the marker disappears and both devices are connected andspatially synced.

Various examples include a method 250 for viewing an augmented realityenvironment. For example, as described herein, a user device is able tojoin to an augmented reality session of the augmented reality device. Asdescribed herein, holograms can be displayed in an augmented realitysession that are spatially aligned to a view of a user device.

The method 250, in one example, includes generating a unique codecorresponding to a user device at 252. As described herein, a unique ARmarker is generated by a user device of a user that desires to join anaugmented reality session (e.g., a shared experience session). Thisuniquely generated AR marker allows an augmented reality device, such asa head-mounted device that is operating in an augmented reality session,to authenticate the user device as described herein. The AR marker canbe generated using any marker generation technique implemented by anaugmented reality marker generator.

The method 250 includes broadcasting information corresponding to theunique code from the user device at 254. For example, a smartphonebroadcasts AR marker information or session request information that isformatted or otherwise configured for detection by augmented realitydevices. That is, the user device broadcasts information on a connectedwireless network that is detectable by augmented reality devices andindicates that the user device desires to join the augmented realitysession. In some examples, the augmented reality device invites the userdevice to join the augmented reality session.

The method 250 includes displaying the unique code on the user device at256. For example, the AR marker associated with the broadcastinformation, and unique to the user device, is displayed on the screenof the user device. The AR maker is displayed for a defined time periodin some examples, such as two minutes. The displayed AR marker allowsfor authentication of the user device to join the augmented realitysession as a trusted device. As such, the user device can have a sharedaugmented reality session with an augmented reality device onceauthenticated.

The method 250 includes, at 258, activating a camera of the augmentedreality device upon receiving the information broadcast from the userdevice, wherein the camera is configured to scan for the unique code.That is, with the received broadcast information, the augmented realitydevice is able to confirm that the displayed AR marker corresponds tothe broadcast information. One or more AR marker matching techniques areused to compare the AR maker to the received information to confirm thematch. For example, characteristics of the AR marker, as contained inthe broadcast information, are identified and confirmed. If the ARmarker does not match the received broadcast information, the userdevice is not authenticated (e.g., the user device did not broadcast theinformation). If the AR marker matches the received broadcastinformation, the user device is authenticated as the device that desiresto join the augmented reality session.

The method 250 includes joining the user device to the augmented realitysession of the augmented reality device at 260 in response to the camerascanning the unique code displayed on the user device by transmittingidentification information to the user device based on the unique code.That is, once the AR marker is confirmed by the segmented realitydevice, an authentication or session acceptance signal is transmitted tothe user device, which includes the connection information for joiningthe augmented reality session. For example, the session ID, connectionID, and/or other information to allow the user device to connect to theaugmented reality session is transmitted.

Thus, various examples include a simple and secure authentication andconnection process for joining an augmented reality session. Theautomated process allows a user device to be quickly identified by anaugmented reality device and spatially aligned shared augmented realitysession thereafter established as described herein.

Exemplary Operating Environment

The present disclosure is operable with a computing apparatus 302according to an embodiment as a functional block diagram 300 in FIG. 3.In one example, components of the computing apparatus 302 can beimplemented as a part of an electronic device according to one or moreembodiments described in this specification. The computing apparatus 302comprises one or more processors 304 which can be microprocessors,controllers or any other suitable type of processors for processingcomputer executable instructions to control the operation of theelectronic device. Platform software comprising an operating system 306or any other suitable platform software can be provided on the apparatus302 to enable application software 308 to be executed on the device.According to an embodiment, viewing of an augmented reality environment310 from a user device 312 can be accomplished by software.

The computing apparatus further includes an augmented reality (AR)session component 324. It should be noted that the AR session component324 can be within one or more of the user device 312, the head worncomputer 108 (shown in FIG. 1), or other components of the variousexamples. The AR session component 324 is configured to performoperations or methods described herein, including, for example, toinitialize, authenticate and/or join the user device 312 (e.g., thesmartphone 102 or the tablet 106) to the head worn computer 108operating as an augmented reality device, such as illustrated in FIGS.2-5.

A memory 314 stores, among other data, one or more applications oralgorithms that include data and executable instructions. Theapplications, when executed by the processor, operate to performfunctionality on the computing device. Exemplary applications includeaugmented reality applications and/or components, such as the AR sessioncomponent 324, for example.

Computer executable instructions can be provided using anycomputer-readable media that are accessible by the computing apparatus302. Computer-readable media may include, for example, computer storagemedia such as the memory 314 and communications media. Computer storagemedia, such as the memory 314, include volatile and non-volatile,removable and non-removable media implemented in any method ortechnology for storage of information such as computer readableinstructions, data structures, program modules or the like. Computerstorage media include, but are not limited to, RAM, ROM, EPROM, EEPROM,flash memory or other memory technology, CD-ROM, digital versatile disks(DVD) or other optical storage, magnetic cassettes, magnetic tape,magnetic disk storage or other magnetic storage devices, or any othernon-transmission medium that can be used to store information for accessby a computing apparatus. In contrast, communication media may embodycomputer readable instructions, data structures, program modules, or thelike in a modulated data signal, such as a carrier wave, or othertransport mechanism. As defined herein, computer storage media do notinclude communication media. Therefore, a computer storage medium shouldnot be interpreted to be a propagating signal per se. Propagated signalsper se are not examples of computer storage media. Although the computerstorage medium (the memory 314) is shown within the computing apparatus302, it will be appreciated by a person skilled in the art, that thestorage may be distributed or located remotely and accessed via anetwork or other communication link (e.g. using a communicationinterface 316).

The computing apparatus 302 can comprise an input/output controller 318configured to output information to one or more input devices 320 andoutput devices 322, for example a display or a speaker, which can beseparate from or integral to the electronic device. The input/outputcontroller 318 can also be configured to receive and process an inputfrom the one or more input devices 320, for example, a keyboard, amicrophone or a touchpad. In one embodiment, the output device 322 canalso act as the input device 320. An example of such a device can be atouch sensitive display. The input/output controller 318 can also outputdata to devices other than the output device 322, e.g. a locallyconnected printing device. In some embodiments, a user can provide inputto the input device(s) 320 and/or receive output from the outputdevice(s) 322.

In some examples, the computing apparatus 302 detects voice input, usergestures or other user actions and provides a natural user interface(NUI). This user input can be used to author electronic ink, viewcontent, select ink controls, play videos with electronic ink overlaysand for other purposes. The input/output controller 318 outputs data todevices other than a display device in some examples, e.g. a locallyconnected printing device.

NUI technology enables a user to interact with the computing apparatus302 in a natural manner, free from artificial constraints imposed byinput devices such as mice, keyboards, remote controls and the like.Examples of NUI technology that are provided in some examples includebut are not limited to those relying on voice and/or speech recognition,touch and/or stylus recognition (touch sensitive displays), gesturerecognition both on screen and adjacent to the screen, air gestures,head and eye tracking, voice and speech, vision, touch, gestures, andmachine intelligence. Other examples of NUI technology that are used insome examples include intention and goal understanding systems, motiongesture detection systems using depth cameras (such as stereoscopiccamera systems, infrared camera systems, red green blue (rgb) camerasystems and combinations of these), motion gesture detection usingaccelerometers/gyroscopes, facial recognition, three dimensional (3D)displays, head, eye and gaze tracking, immersive augmented reality andvirtual reality systems and technologies for sensing brain activityusing electric field sensing electrodes (electro encephalogram (EEG) andrelated methods).

The functionality described herein can be performed, at least in part,by one or more hardware logic components. According to an embodiment,the computing apparatus 302 is configured by the program code whenexecuted by the processor(s) 304 to execute the embodiments of theoperations and functionality described. Alternatively, or in addition,the functionality described herein can be performed, at least in part,by one or more hardware logic components. For example, and withoutlimitation, illustrative types of hardware logic components that can beused include Field-programmable Gate Arrays (FPGAs),Application-specific Integrated Circuits (ASICs), Program-specificStandard Products (ASSPs), System-on-a-chip systems (SOCs), ComplexProgrammable Logic Devices (CPLDs), Graphics Processing Units (GPUs).

At least a portion of the functionality of the various elements in thefigures can be performed by other elements in the figures, or an entity(e.g., processor, web service, server, application program, computingdevice, etc.) not shown in the figures.

Although described in connection with an exemplary computing systemenvironment, examples of the disclosure are capable of implementationwith numerous other general purpose or special purpose computing systemenvironments, configurations, or devices.

Examples of well-known computing systems, environments, and/orconfigurations that can be suitable for use with aspects of thedisclosure include, but are not limited to, mobile or portable computingdevices (e.g., smartphones), personal computers, server computers,hand-held (e.g., tablet) or laptop devices, multiprocessor systems,gaming consoles or controllers, microprocessor-based systems, set topboxes, programmable consumer electronics, mobile telephones, mobilecomputing and/or communication devices in wearable or accessory formfactors (e.g., watches, glasses, headsets, or earphones), network PCs,minicomputers, mainframe computers, distributed computing environmentsthat include any of the above systems or devices, and the like. Ingeneral, the disclosure is operable with any device with processingcapability such that it can execute instructions such as those describedherein. Such systems or devices can accept input from the user in anyway, including from input devices such as a keyboard or pointing device,via gesture input, proximity input (such as by hovering), and/or viavoice input.

Examples of the disclosure may be described in the general context ofcomputer-executable instructions, such as program modules, executed byone or more computers or other devices in software, firmware, hardware,or a combination thereof. The computer-executable instructions can beorganized into one or more computer-executable components or modules.Generally, program modules include, but are not limited to, routines,programs, objects, components, and data structures that performparticular tasks or implement particular abstract data types. Aspects ofthe disclosure can be implemented with any number and organization ofsuch components or modules. For example, aspects of the disclosure arenot limited to the specific computer-executable instructions or thespecific components or modules illustrated in the figures and describedherein. Other examples of the disclosure can include differentcomputer-executable instructions or components having more or lessfunctionality than illustrated and described herein.

In examples involving a general-purpose computer, aspects of thedisclosure transform the general-purpose computer into a special-purposecomputing device when configured to execute the instructions describedherein.

Other examples include:

-   -   A system for augmented reality viewing, the system comprising:    -   a memory associated with a computing device, the memory        including an augmented reality session component; and    -   a processor that executes the augmented reality session        component to:    -   generate a unique code corresponding to a user device;    -   broadcast information corresponding to the unique code from the        user device, the information configured to be received by an        augmented reality device;    -   display the unique code on the user device; and    -   join the user device to an augmented reality session of the        augmented reality device using received identification        information from the augmented reality device and based at least        in part on the displayed unique code.

Other examples include:

-   -   A computerized method for viewing an augmented reality        environment, the computerized method comprising:    -   generating a unique code at a user device and displaying the        unique code on the user device;    -   broadcasting information formatted for reception by an augmented        reality device, the broadcast information based at least in part        on the unique code; and    -   automatically joining the user device to an augmented reality        session of the augmented reality device in response to the        augmented reality device scanning the unique code displayed on        the user device, at least one hologram in the augmented reality        session being spatially aligned to a view of the user device.

Other examples include:

-   -   One or more computer storage media having computer-executable        instructions for viewing an augmented reality environment that,        upon execution by a processor, cause the processor to at least:    -   generate a unique code corresponding to a user device;    -   broadcast information corresponding to the unique code from the        user device, the information configured to be received by an        augmented reality device;    -   display the unique code on the user device; and    -   join the user device to an augmented reality session of the        augmented reality device using received identification        information from the augmented reality device and based at least        in part on the displayed unique code.

Alternatively, or in addition to the other examples described herein,examples include any combination of the following:

-   -   spatially aligning a hologram being displayed in the augmented        reality session to a view of the user device.    -   activating the camera of the augmented reality device upon        receiving the information broadcast from the user device, the        camera configured to scan for the unique code.    -   wherein an initiation of a session joining application on the        user device triggers the activation of the camera of the        augmented reality device based at least in part on a        broadcasting of a request to identify bundle identification (ID)        sessions on a same wireless network as the user device.    -   wherein the augmented reality device transmits identification        information to the user device based on the unique code to        confirm a match with the displayed unique code.    -   wherein the augmented reality device comprises a HoloLens®        device, the user device comprises one of a mobile phone or a        tablet device, and the unique code comprises an augmented        reality marker, the mobile phone or the tablet device is        configured to generate and display the augmented reality marker        and the HoloLens® device is configured to scan for the augmented        reality marker and match the augmented reality marker to the        user device based at least in part on the information broadcast        from the user device.    -   receiving spatial information from the augmented reality device        and display an augmented reality session having at least one        hologram being viewing by the augmented reality device with the        hologram aligned spatially within the augmented reality space as        viewed by the augmented reality device.    -   changing an augmented reality space of the augmented reality        session based on a control command received from the user        device.

Any range or device value given herein can be extended or alteredwithout losing the effect sought, as will be apparent to the skilledperson.

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described above.Rather, the specific features and acts described above are disclosed asexample forms of implementing the claims.

It will be understood that the benefits and advantages described abovecan relate to one embodiment or can relate to several embodiments. Theembodiments are not limited to those that solve any or all of the statedproblems or those that have any or all of the stated benefits andadvantages. It will further be understood that reference to ‘an’ itemrefers to one or more of those items.

The embodiments illustrated and described herein as well as embodimentsnot specifically described herein but within the scope of aspects of theclaims constitute exemplary means for training a neural network. Theillustrated one or more processors 304 together with the computerprogram code stored in memory 314 constitute exemplary processing meansfor using and/or training neural networks.

The term “comprising” is used in this specification to mean includingthe feature(s) or act(s) followed thereafter, without excluding thepresence of one or more additional features or acts.

In some examples, the operations illustrated in the figures can beimplemented as software instructions encoded on a computer readablemedium, in hardware programmed or designed to perform the operations, orboth. For example, aspects of the disclosure can be implemented as asystem on a chip or other circuitry including a plurality ofinterconnected, electrically conductive elements.

The order of execution or performance of the operations in examples ofthe disclosure illustrated and described herein is not essential, unlessotherwise specified. That is, the operations can be performed in anyorder, unless otherwise specified, and examples of the disclosure caninclude additional or fewer operations than those disclosed herein. Forexample, it is contemplated that executing or performing a particularoperation before, contemporaneously with, or after another operation iswithin the scope of aspects of the disclosure.

When introducing elements of aspects of the disclosure or the examplesthereof, the articles “a,” “an,” “the,” and “said” are intended to meanthat there are one or more of the elements. The terms “comprising,”“including,” and “having” are intended to be inclusive and mean thatthere can be additional elements other than the listed elements. Theterm “exemplary” is intended to mean “an example of” The phrase “one ormore of the following: A, B, and C” means “at least one of A and/or atleast one of B and/or at least one of C.”

Having described aspects of the disclosure in detail, it will beapparent that modifications and variations are possible withoutdeparting from the scope of aspects of the disclosure as defined in theappended claims. As various changes could be made in the aboveconstructions, products, and methods without departing from the scope ofaspects of the disclosure, it is intended that all matter contained inthe above description and shown in the accompanying drawings shall beinterpreted as illustrative and not in a limiting sense.

What is claimed is:
 1. A system for augmented reality viewing, thesystem comprising: a memory associated with a computing device, thememory including an augmented reality session component; and a processorthat executes the augmented reality session component to: generate aunique code corresponding to a user device; broadcast informationcorresponding to the unique code from the user device, the informationconfigured to be received by an augmented reality device; display theunique code on the user device; and join the user device to an augmentedreality session of the augmented reality device using receivedidentification information from the augmented reality device and basedat least in part on the displayed unique code.
 2. The system of claim 1,wherein the processor further executes the augmented reality sessioncomponent to spatially align a hologram being displayed in the augmentedreality session to a view of the user device.
 3. The system of claim 1,wherein the augmented reality device comprises a camera and theprocessor further executes the augmented reality session component toactivate the camera of the augmented reality device upon receiving theinformation broadcast from the user device, the camera configured toscan for the unique code.
 4. The system of claim 3, wherein aninitiation of a session joining application on the user device triggersthe activation of the camera of the augmented reality device based atleast in part on a broadcasting of a request to identify bundleidentification (ID) sessions on a same wireless network as the userdevice.
 5. The system of claim 3, wherein the augmented reality devicetransmits identification information to the user device based on theunique code to confirm a match with the displayed unique code.
 6. Thesystem of claim 1, wherein the augmented reality device comprises aHoloLens® device, the user device comprises one of a mobile phone or atablet device, and the unique code comprises an augmented realitymarker, the mobile phone or the tablet device is configured to generateand display the augmented reality marker and the HoloLens® device isconfigured to scan for the augmented reality marker and match theaugmented reality marker to the user device based at least in part onthe information broadcast from the user device.
 7. The system of claim1, wherein the processor further executes the augmented reality sessioncomponent to receive spatial information from the augmented realitydevice and display an augmented reality session having at least onehologram being viewing by the augmented reality device with the hologramaligned spatially within the augmented reality space as viewed by theaugmented reality device.
 8. The system of claim 1, wherein theprocessor further executes the augmented reality session component tochange an augmented reality space of the augmented reality session basedon a control command received from the user device.
 9. A computerizedmethod for viewing an augmented reality environment, the computerizedmethod comprising: generating a unique code at a user device anddisplaying the unique code on the user device; broadcasting informationformatted for reception by an augmented reality device, the broadcastinformation based at least in part on the unique code; and automaticallyjoining the user device to an augmented reality session of the augmentedreality device in response to the augmented reality device scanning theunique code displayed on the user device, at least one hologram in theaugmented reality session being spatially aligned to a view of the userdevice.
 10. The computerized method of claim 9, further comprisingautomatically activating a camera of the augmented reality device uponreceiving the information broadcast from the user device, the cameraconfigured to scan for the unique code.
 11. The computerized method ofclaim 10, and further comprising, in response to an initiation of asession joining application on the user device, automatically triggeringthe activation of the camera of the augmented reality device based atleast in part on a broadcasting of a request, from the user device, toidentify bundle identification (ID) sessions on a same wireless networkas the user device.
 12. The computerized method of claim 9, wherein theaugmented reality device comprises a HoloLens® device, the user devicecomprises one of a mobile phone or a tablet device, and the unique codecomprises an augmented reality marker, the mobile phone or the tabletdevice is configured to generate and display the augmented realitymarker and the HoloLens® device is configured to scan for the augmentedreality marker and further comprising matching the augmented realitymarker to the user device based at least in part on the informationbroadcast from the user device.
 13. The computerized method of claim 9,receiving, at the user device, spatial information from the augmentedreality device and displaying the augmented reality session having atleast one hologram being viewing by the augmented reality device withthe hologram aligned spatially within the augmented reality space asviewed by the augmented reality device.
 14. The computerized method ofclaim 9, further comprising, after joining the user device to theaugmented reality session, receiving one or more control commands fromthe user device that changes the augmented reality space.
 15. Thecomputerized method of claim 9, further comprising recording theaugmented reality session at the user device.
 16. One or more computerstorage media having computer-executable instructions for viewing anaugmented reality environment that, upon execution by a processor, causethe processor to at least: generate a unique code corresponding to auser device; broadcast information corresponding to the unique code fromthe user device, the information configured to be received by anaugmented reality device; display the unique code on the user device;and join the user device to an augmented reality session of theaugmented reality device using received identification information fromthe augmented reality device and based at least in part on the displayedunique code.
 17. The one or more computer storage media of claim 16having further computer-executable instructions that, upon execution bya processor, cause the processor to at least spatially align a hologrambeing displayed in the augmented reality session to a view of the userdevice.
 18. The one or more computer storage media of claim 16, whereinthe augmented reality device comprises a camera and having furthercomputer-executable instructions that, upon execution by a processor,cause the processor to at least activate the camera of the augmentedreality device upon receiving the information broadcast from the userdevice, the camera configured to scan for the unique code.
 19. The oneor more computer storage media of claim 16, wherein an initiation of asession joining application on the user device triggers the activationof the camera of the augmented reality device based at least in part ona broadcasting of a request to identify bundle identification (ID)sessions on a same wireless network as the user device.
 20. The one ormore computer storage media of claim 16 having furthercomputer-executable instructions that, upon execution by a processor,cause the processor to at least receive spatial information from theaugmented reality device and display an augmented reality session havingat least one hologram being viewing by the augmented reality device withthe hologram aligned spatially within the augmented reality space asviewed by the augmented reality device, and change an augmented realityspace of the augmented reality session based on a control commandreceived from the user device.